out_influxdb: documentation for tag prefix stripping#1468
out_influxdb: documentation for tag prefix stripping#1468ueli-g wants to merge 8 commits intofluent:masterfrom
Conversation
Signed-off-by: Ueli Graf <ueli@9tlabs.com>
Signed-off-by: Ueli Graf <ueli@9tlabs.com>
|
@fluent/chronosphere-tech-writers should we be ensuring we add YAML config as well now always? |
|
@esmerel can you review please, but leave open as code PR fluent/fluent-bit#9427 must merge first and is still open. |
|
@ueli-g can you resolve the conflicts and provide YAML config example (if you need help with that, happy to provide)? I also nudged you on the code PR fluent/fluent-bit#9427 |
Signed-off-by: Lynette Miles <6818907+esmerel@users.noreply.github.com>
|
I fixed the merge conflicts and made review updates while I was fixing it since I was already in there. |
Signed-off-by: Lynette Miles <6818907+esmerel@users.noreply.github.com>
patrick-stephens
left a comment
There was a problem hiding this comment.
Config example needs updating to latest standard.
Signed-off-by: Ueli Graf <ueli@9tlabs.com>
|
@eschabell thank you for the review. I added a yaml config example as requested and added more information on the behaviour. please let me know if there is anything else I can do to move this forward. |
@patrick-stephens can you please have another look and let me know if this is more appropriate? |
|
@ueli-g would you mind sorting out the conflicts now showing up for this PR? |
Signed-off-by: Ueli Graf <ueli@9tlabs.com>
|
Warning Rate limit exceeded
Your organization is not enrolled in usage-based pricing. Contact your admin to enable usage-based pricing to continue reviews beyond the rate limit, or try again in 54 minutes and 11 seconds. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughDocumentation for a new Changes
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~5 minutes Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
pipeline/outputs/influxdb.md (1)
154-164: Consider clarifying the input configuration in the example.All four inputs use
name: cpueven though the tags suggest different hardware types (gpu). While this is technically valid and simplifies the example to focus on routing and prefix stripping, it might be slightly confusing. Consider adding a brief comment explaining that the same input plugin is used with different tags for demonstration purposes, or acknowledging that in a real scenario, gpu tags might come from different sources.💡 Optional clarification
You could add a comment above the inputs section like:
```yaml pipeline: inputs: + # Using cpu plugin with different tags to demonstrate routing and prefix stripping - name: cpu tag: cpu.oneOr in prose before the example:
-When collecting data from many inputs into many buckets, it can be helpful to remove a common prefix using `Strip_prefix`. +When collecting data from many inputs into many buckets, it can be helpful to remove a common prefix using `Strip_prefix`. The following example uses the same input plugin with different tags to demonstrate how prefix stripping works with pattern-based routing.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@pipeline/outputs/influxdb.md` around lines 154 - 164, The example under pipeline.inputs uses name: cpu for all four entries which can confuse readers because tags include gpu; update the example to either (a) add a brief inline comment above the inputs (referencing pipeline.inputs and the name: cpu entries) explaining that the cpu plugin is intentionally reused with different tags for demonstration of routing/prefix stripping, or (b) change the relevant entries to use a more representative name (e.g., name: gpu) for gpu.* tags so tag names and input plugin names align; ensure the comment or rename clearly references the tag values (cpu.one, cpu.two, gpu.one, gpu.two) so readers understand the intent.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@pipeline/outputs/influxdb.md`:
- Line 25: The documentation shows an inconsistency for the option strip_prefix:
the description says it defaults to an empty string but the Default column shows
"_none_"; update the Default column entry for strip_prefix to an explicit
empty-string representation (e.g., `""`) so it matches the textual description
and makes the default behavior clear; locate the table row referencing
strip_prefix and change the Default cell to `""` while leaving the description
and other cells unchanged.
- Around line 186-213: The fluent-bit.conf influxdb OUTPUT examples are missing
the Host and Port parameters shown in the YAML example; update both influxdb
OUTPUT blocks (the ones matching "cpu*" and "gpu*") to include Host and Port
entries (e.g., Host and Port keys alongside Name, Match, Bucket, and
Strip_prefix) so the two examples are consistent and clearly show the connection
endpoint settings.
---
Nitpick comments:
In `@pipeline/outputs/influxdb.md`:
- Around line 154-164: The example under pipeline.inputs uses name: cpu for all
four entries which can confuse readers because tags include gpu; update the
example to either (a) add a brief inline comment above the inputs (referencing
pipeline.inputs and the name: cpu entries) explaining that the cpu plugin is
intentionally reused with different tags for demonstration of routing/prefix
stripping, or (b) change the relevant entries to use a more representative name
(e.g., name: gpu) for gpu.* tags so tag names and input plugin names align;
ensure the comment or rename clearly references the tag values (cpu.one,
cpu.two, gpu.one, gpu.two) so readers understand the intent.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: d0e3e2e3-0f34-4198-bac2-fddc13977b39
📒 Files selected for processing (1)
pipeline/outputs/influxdb.md
| | `org` | InfluxDB organization name where the bucket is (v2 only). | `fluent` | | ||
| | `port` | TCP port of the target InfluxDB service. | `8086` | | ||
| | `sequence_tag` | The name of the tag whose value is incremented for consecutive simultaneous events. | _none_ | | ||
| | `strip_prefix` | String prefix to be removed from the front of `tag` when writing InfluxDB measurement names. `strip_prefix` is removed only if it matches exactly at the beginning of and is strictly shorter than `tag` (`tag` length will not be reduced to zero if `strip_prefix` and `tag` are identical strings). `strip_prefix` defaults to empty string, which does nothing. | _none_ | |
There was a problem hiding this comment.
Inconsistency between Default column and description.
The Default column shows "none" but the description explicitly states "strip_prefix defaults to empty string, which does nothing." These should be consistent. Consider changing the Default column to "" (empty string) to match the described behavior.
📝 Proposed fix
-| `strip_prefix` | String prefix to be removed from the front of `tag` when writing InfluxDB measurement names. `strip_prefix` is removed only if it matches exactly at the beginning of and is strictly shorter than `tag` (`tag` length will not be reduced to zero if `strip_prefix` and `tag` are identical strings). `strip_prefix` defaults to empty string, which does nothing. | _none_ |
+| `strip_prefix` | String prefix to be removed from the front of `tag` when writing InfluxDB measurement names. `strip_prefix` is removed only if it matches exactly at the beginning of and is strictly shorter than `tag` (`tag` length will not be reduced to zero if `strip_prefix` and `tag` are identical strings). `strip_prefix` defaults to empty string, which does nothing. | `""` |📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| | `strip_prefix` | String prefix to be removed from the front of `tag` when writing InfluxDB measurement names. `strip_prefix` is removed only if it matches exactly at the beginning of and is strictly shorter than `tag` (`tag` length will not be reduced to zero if `strip_prefix` and `tag` are identical strings). `strip_prefix` defaults to empty string, which does nothing. | _none_ | | |
| | `strip_prefix` | String prefix to be removed from the front of `tag` when writing InfluxDB measurement names. `strip_prefix` is removed only if it matches exactly at the beginning of and is strictly shorter than `tag` (`tag` length will not be reduced to zero if `strip_prefix` and `tag` are identical strings). `strip_prefix` defaults to empty string, which does nothing. | `""` | |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@pipeline/outputs/influxdb.md` at line 25, The documentation shows an
inconsistency for the option strip_prefix: the description says it defaults to
an empty string but the Default column shows "_none_"; update the Default column
entry for strip_prefix to an explicit empty-string representation (e.g., `""`)
so it matches the textual description and makes the default behavior clear;
locate the table row referencing strip_prefix and change the Default cell to
`""` while leaving the description and other cells unchanged.
Signed-off-by: Ueli Graf <ueli@9tlabs.com>
Signed-off-by: Ueli Graf <ueli@9tlabs.com>
|
@edsiper thank you for your review, updated accordingly. This has remained open for a while now, please let me know if there is anything that I can do or that I am missing to help move this and the corresponding feature in fluent-bit along. |
| | `org` | InfluxDB organization name where the bucket is (v2 only). | `fluent` | | ||
| | `port` | TCP port of the target InfluxDB service. | `8086` | | ||
| | `sequence_tag` | The name of the tag whose value is incremented for consecutive simultaneous events. | _none_ | | ||
| | `strip_prefix` | String prefix to be removed from the front of `tag` when writing InfluxDB measurement names. `strip_prefix` is removed only if it matches exactly at the beginning of and is strictly shorter than `tag` (`tag` length will not be reduced to zero if `strip_prefix` and `tag` are identical strings). | _none_ | |
| `Auto_Tags=On` in this example causes an error, because every parsed field value type is `string`. The best usage of this option in metrics like record where one or more field value isn't `string` typed. | ||
|
|
||
|
|
||
| ### Prefix stripping |
There was a problem hiding this comment.
@ueli-g please format all yaml and classic config example code with 2 spaced indents (currently using 4?).
Documentation to go with fluent/fluent-bit#9427
This removes a defined prefix from measurement names which might otherwise be shared between many measurements in the same data bucket. Compatibility is not affected, default behavior (with documented default strip_prefix "") is to strip zero character from the front which identical to the existing mechanism.
When writing to a range of different buckets, routing to the corresponding out_influxdb instances happens on tag matches. This change allows to match on tag prefixes, but strip them from the measurement name. This avoids having identical prefixes for all measurement names in the same data bucket.
To achieve this, read from char tag[] with an offset when writing the measurement name, provided the prefix matches the tag completely and the overlap is at most tag_length - 1 characters.
Summary by CodeRabbit
Release Notes
strip_prefixconfiguration parameter for InfluxDB output to remove exact prefixes from measurement names, simplifying event tag normalization.